Conversation
Refs #414. wiki-snapshot fetches the Markdown mirror once into a local directory with per-document hashes and a snapshot hash. Runs with knowledge wiki require a snapshot and record its identity. Fetching goes through curl because the mirror returns 403 to Python's HTTP client.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Refs #414. Stacked on #464 (base is
feat/414-pi-devin-adapters); retarget tomainwhen #464 merges.The
wikiknowledge level had no data source: there is no local wiki, only the Workshop.codes wiki online and its Markdown mirror atmd.wrightkit.dev. This adds a pinned snapshot of the mirror.agent_bench.py wiki-snapshot [--dir DIR]readsmanifest.json, fetches each article once, and writesarticles/,index.md,NOTICE.txt, andSNAPSHOT.json(per-document SHA-256, manifest hash, and asnapshotSha256over the sorted slug/hash pairs). It refuses to write into a directory that already holds a snapshot, and rejects slugs that are not[A-Za-z0-9_-]+.knowledgewikinow requires--wiki-dirto hold aSNAPSHOT.json, links it as./wiki, and recordsenvironment.wiki(source, fetch time, hash, document count) in the result, so results cite the exact documentation the agent saw.curl: the mirror returns 403 to Python's HTTP client (urllib, with or without a custom User-Agent) and 200 to curl.Scope of the corpus: the mirror's manifest currently lists 24 articles (single-action and value notes, a few reference tables, and tutorials), so the
wikilevel tests a small, partial reference. That is what the wiki has today, and the benchmark should say so instead of implying full API documentation.Licensing: the content is the Workshop.codes wiki and follows its Terms of Service, as
workshop-md-converterstates. The snapshot is local, written to~/.cacheby default, and never committed;NOTICE.txtsays so.Verification: 29 tests pass. The snapshot tests run against a local HTTP server (stable identity across two snapshots, pinned directories, unsafe slugs refused, wiki level requires a snapshot); another test checks that the harness links the snapshot, records its identity, and does not count
./wikias an unsafe edit. I did not run a real snapshot; that downloads the articles to a local directory and is left to whoever runs the evaluation.